Object detecting method and object detecting apparatus and intruding object monitoring apparatus employing the object detecting method

ABSTRACT

An object detecting method, object detecting apparatus and intruding object monitoring apparatus employing the object detecting method. A predetermined monitor area having no object to be detected is imaged by an imaging device and reference background sequential images are registered in advance in a storage unit. When monitoring, the predetermined monitor area is imaged by the imaging device while at the same time sequentially outputting the images frame by frame from the imaging device, a reference background image corresponding to the image from the imaging device is selected from the reference background sequential images, subtraction processing is carried out between the image from the imaging device and a corresponding reference background image, and the object detecting processing is carried out based on the result of subtraction processing.

BACKGROUND OF THE INVENTION

[0001] The present invention relates to the detection of an object usingan imaging device, or in particular to an object detecting method and anobject detecting apparatus and an intruding object monitoring apparatusfor automatically detecting an object intruding into the monitoringvisual field by scanning a predetermined monitor area with the imagingdevice while changing the zoom setting of the zoom lens of the imagingdevice, changing the imaging direction of the imaging device or movingthe imaging device, thereby changing the monitoring visual field.

[0002] An intruding object detecting apparatus using an imaging devicesuch as a camera has been widely used. In recent years, however, demandhas arisen for an intruding object detecting apparatus in which anintruding object such as a person or an automobile (vehicle) intrudinginto a monitoring visual field is detected not by a person watching theimage displayed on a monitor, but automatically by the image signalinput from the imaging device and a predetermined announcement or alarmis issued by appropriate means.

[0003] In order to realize such an intruding object detecting apparatus,first, an inputted image from the imaging device is compared with animage in which an object to be detected is not present, such as areference background image, an input image which has been previouslyobtained or the like, through image processing and a differencetherebetween is detected. For example, an input image and a referencebackground image in which an object to be detected is not present arecompared with each other, a difference in the brightness or luminancevalue for each pixel is detected and an area of pixels that have largedifference values is detected as an intruding object. This method forobtaining a difference is known as “subtraction method” and has beenwidely used.

[0004] The processing by the subtraction method will be explained withreference to FIG. 9. FIG. 9 is a diagram for explaining the principle ofprocessing the object detection according to the subtraction method.Reference numeral 901 designates an input image, numeral 902 a referencebackground image, numeral 903 a difference image, numeral 904 abinarized image of the difference image 903, numeral 905 a subtractor,numeral 906 a binarizer, numeral 907 an person-like object picked up inthe input image 901, numeral 908 an area generated by the subtraction,and numeral 909 an image of a cluster of the brightness values “255”binarized from the area 908 generated by the subtraction.

[0005] In FIG. 9, the subtractor 905 calculates the difference of thebrightness value or luminance value between the input image 901 and thereference background ground image 902 for each pixel, and outputs thedifference image 903. In the difference image 903, a person-like object907 picked up in the input image 901, for example, is applied to thebinarizer 906 as an area 908 where the difference has been generatedbetween the reference background image 902 and the input image 901.

[0006] The binarizer 906 produces a binarized image 904 by determiningthe brightness value of each pixel less than a predetermined thresholdvalue Th as “0” and the brightness value of the pixel not less than thethreshold value Th as “255” (the brightness value of one pixelcalculated as 8 bits). The person-like object 907 picked up in the inputimage 901 in this way is calculated as an area 908 for which adifference is generated by the subtractor 905, and detected by thebinarizer 906 as an image 909 indicating a cluster of pixels having thebrightness value of “255”.

SUMMARY OF THE INVENTION

[0007] As described above, the subtraction method requires thepreparation of a reference background image where an intruding object tobe detected is not present. In the case where the viewing angle or theviewing direction become different from the reference background imageas a result of changing the zoom setting of the zoom lens or the imagingdirection of the imaging device or moving the imaging device to changethe visual field of the camera, however, the reference background imageprepared in advance becomes of no use, thereby posing the problem thatthe subtraction method is not applicable in this case. Even if anattempt is made to prepare a new reference background image, the objectdetection becomes impossible to do during the preparation. In theconventional intruding object detecting method according to thesubtraction method, therefore, it is practically impossible to detect anintruding object while changing the zoom setting or the imagingdirection of the imaging device or moving the imaging device itself.

[0008] Accordingly, the object of the present invention is to obviatethe disadvantages of the prior art described above, and to provide anobject detecting method, an object detecting apparatus and an intrudingobject monitoring apparatus which are reliable and capable of detectingan object such as an intruding object even in the case where the zoomsetting or the imaging direction of the imaging device is changed.

[0009] According to one aspect of the invention, there is provided anobject detecting method for detecting an object in a predeterminedmonitor area, comprising the steps of:

[0010] imaging a plurality of different areas in the predeterminedmonitor area having no object to be detected at a predetermined time byan imaging device and registering a plurality of images corresponding tothe different areas having no image of the object to be detected;

[0011] at a different time from the predetermined time, imaging thepredetermined monitor area by the imaging device;

[0012] comparing an image from the imaging device with a correspondingone of the plurality of registered images; and

[0013] detecting the object to be detected based on a result of thecomparison.

[0014] In one embodiment, the plurality of registered images having noimage of the object to be detected are reference background sequentialimages (a group of reference background images) and the step ofcomparing is subtraction processing between the image from the imagingdevice and the corresponding one of the reference background sequentialimages.

[0015] The imaging visual field of the imaging device in imagingoperation undergoes a change every moment. According to the presentmethod of object detection, however, the subtraction processing can becarried out in such a manner that a reference background image mostcoincident with the present imaging visual field (or the image picked upin the particular imaging visual field) can be selected from thereference background sequential images.

[0016] According to a preferable feature of the invention, this objectdetecting method includes the step of detecting a displacement betweenthe image from the imaging device and a corresponding referencebackground image, then correcting the image from the imaging device inaccordance with the detected displacement, wherein the subtractionprocessing is carried out between the corrected image and thecorresponding reference background image.

[0017] Specifically, in the case where the imaging visual field ischanged by moving the imaging device, the imaging device trembles anddevelops a displacement, so that there also occurs a displacementbetween the image from the imaging device and a reference backgroundimage. This displacement is detected erroneously as an object at thetime of object detection, i.e. in the subtraction and subsequentprocess. In order to remove this displacement, the step of detecting thedisplacement preferably includes the step of applying a templatematching between the image from the imaging device and the correspondingreference background image to detect the displacement.

[0018] According to an embodiment, in the template matching step, aplurality of sections are set on the reference background image, and theimage of each section is used as a template which is used for templatematching with the image from the imaging device, and the average of thedisplacements detected is used as the displacement.

[0019] Another preferable feature of the invention is that the objectdetecting method described above comprises the step of detecting a framedisplacement between the frame of the image from the imaging device andthe frame of the corresponding reference background image, wherein whenthere is any frame displacement, another reference background image isselected.

[0020] In other words, it is critical to select a proper referencebackground image when carrying out the subtraction process. In the casewhere a proper reference background image is not selected from thereference background sequential images and a frame displacement occurs,the background portion would be displaced between the image from theimaging device and the selected reference background image, and theparticular displacement would be detected erroneously as an object atthe time of object detection, i.e. in the subtraction and subsequentsteps. In order to remove this displacement, the frame displacementdetecting step preferably includes the step of applying a templatematching between the image from the imaging device and the correspondingreference background image to detect the frame displacement.

[0021] According to an embodiment, in the template matching step, aplurality of sections are set on the corresponding reference backgroundimage, and using the image of each section as a template, the templatematching is carried out with the image from the imaging device. Inaccordance with the frame displacement information thus detected, areference background image of the frame temporally preceding orsucceeding to the frame of the reference background image is selected.

[0022] As an alternative, preferably, the frame displacement detectingstep is such that the frame displacement is corrected based on at leastone of the position of the imaging device and the imaging visual fieldinformation of the imaging device.

[0023] According to an embodiment, a specific position of the imagingdevice and a specific frame of the reference background imagecorresponding to the specific position are set in association with eachother in advance, and when the imaging device arrives at the specificposition, the frame displacement is corrected using the referencebackground image of the specific frame.

[0024] According to an embodiment, the imaging visual field informationcontains a specific object within the predetermined monitor visual fieldas a mark, and this mark and the reference background image of aspecific frame corresponding to the mark are set in association witheach other in advance. When the imaging device picks up the image of themark, the frame displacement is corrected using the reference backgroundimage of the specific frame.

[0025] According to a further preferable feature of the invention, theobject detecting method includes the step of updating the referencebackground sequential images to update at least one of the referencebackground sequential images.

[0026] According to an embodiment, the update step functions in such amanner that when an object to be detected is not detected in the imagefrom the imaging device in the object detection processing step, thecorresponding reference background image is updated with the image.

[0027] According to another aspect of the invention, there is providedan object detecting method for detecting an object, comprising the stepsof:

[0028] imaging a plurality of different areas in a predetermined monitorarea at a predetermined time by the imaging device in accordance with apredetermined scanning pattern, and registering respective referencebackground sequential images corresponding to the different areas in astorage device;

[0029] at a different time from the predetermined time, imaging thepredetermined monitor area with the imaging device substantially inaccordance with the predetermined scanning pattern in synchronism withthe operation of reading the registered reference background sequentialimages from the storage device;

[0030] carrying out the subtraction processing between the images fromthe imaging device and the reference background sequential images readout; and

[0031] carrying out object detecting processing based on a result of thesubtraction processing.

[0032] The predetermined scanning pattern includes a predeterminedchronological change of at least one or a combination of two or more of,for example, the zoom ratio of the zoom lens of the imaging device, theimaging direction and the travel route of the imaging device.

[0033] According to an embodiment, the predetermined scanning pattern isobtained by the imaging device moving on a predetermined speed profilealong a predetermined travel route.

[0034] According to another embodiment, the predetermined scanningpattern includes a pattern in which the position of the imaging deviceis fixed while the zoom ratio and the imaging direction of the imagingdevice periodically change.

[0035] Another preferable feature of the invention is that the referencebackground sequential images are a set of images obtained by scanningthe predetermined monitor area having no object with the imaging deviceaccording to the predetermined scanning pattern and sampling the frameimages from the imaging device at predetermined sampling intervals. Inthe registration step, each reference background frame image of thereference background sequential images is registered with a frame numberassigned in order of imaging or sampling. In the imaging step, on theother hand, the imaging operation of the imaging device and theoperation of reading the reference background image from theregistration device are carried out in synchronism with each other usingthe frame number, so that a reference background image corresponding tothe image from the imaging device is selected from the referencebackground sequential images.

[0036] According to an embodiment, the frame number of the correspondingreference background image is calculated from the relation between thetime elapsed from the start of monitoring to the present point in timeand the predetermined sampling interval, and the imaging operation ofthe imaging device and the operation of reading the reference backgroundimage from the registration device are synchronized with each otherusing the calculated frame number.

[0037] Another preferable feature of the invention lies in the step ofdetecting a frame displacement between the image from the imaging deviceand the selected reference background image, wherein in the presence ofa frame displacement, a different reference background image isselected.

[0038] Still another preferable feature of the invention lies in thestep of detecting a spatial displacement between the image from theimaging device and the reference background image that has been read andcorrecting the read image in accordance with the detected spatialdisplacement, wherein the subtraction processing is carried out usingthe corrected image.

[0039] According to still another aspect of the invention, there isprovided an object detecting apparatus for detecting an object locatedwithin a predetermined monitor area, comprising:

[0040] an imaging device;

[0041] an image input interface connected to the imaging device forconverting a video signal of the imaging device to image data;

[0042] a processing unit including a central processing unit and amemory for processing the image data; and

[0043] a bus for interconnecting the image input interface and theprocessing unit;

[0044] wherein in order to detect the object, the processing unitcontrols the object detecting apparatus such that:

[0045] images from the imaging device imaging the predetermined monitorarea having no object to be detected are stored sequentially in thememory;

[0046] an image from the imaging device imaging the predeterminedmonitor area is sequentially inputted to the processing unit;

[0047] an image having no object to be detected corresponding to theinputted image is read out of the image memory; and

[0048] the input image and the read image having no object to bedetected are compared and based on a result of comparison, objectdetection processing is carried out.

[0049] An embodiment of the invention includes a zoom lens control unitconnected to the bus for changing the zoom ratio of the zoom lens of theimaging device and a pan and tilt head control unit for changing theimaging direction of the imaging device.

[0050] According to another embodiment, the imaging device is mounted ona moving device.

[0051] According to an embodiment, the moving device includes a mobileunit.

[0052] According to another embodiment, the moving device includes a panand tilt head.

[0053] According to yet another aspect of the invention, there isprovided an object detecting apparatus for detecting an object within apredetermined monitor area, comprising:

[0054] an imaging device;

[0055] an image input interface connected to the imaging device forconverting the video signal from the imaging device to image data;

[0056] a processing unit including a central processing unit and amemory for processing the image data; and

[0057] a bus for interconnecting the image input interface and theprocessing unit;

[0058] wherein in order to detect the object, the processing unitcontrols the object detecting apparatus such that:

[0059] an image from the imaging device imaging the predeterminedmonitor area having no object to be detected is recorded in the imagememory beforehand as a reference background image of referencebackground sequential images;

[0060] an image from the imaging device imaging the predeterminedmonitor area in accordance with a predetermined scanning pattern issequentially input to the processing unit;

[0061] a reference background image is read out of the referencebackground sequential images from the image memory in synchronism withthe imaging of the input images; and

[0062] a difference in a pixel value for each pixel between the inputimages and the read reference background image is calculated, and anarea associated with a large difference value is detected as the object.

[0063] According to a further aspect of the invention, there is provideda monitoring apparatus for monitoring an object intruding into apredetermined monitor area, comprising:

[0064] an imaging device;

[0065] an image input interface connected to the imaging device forconverting a video signal from the imaging device to image data;

[0066] a processing unit including a central processing unit and amemory for processing the image data;

[0067] a monitor; and

[0068] a bus for interconnecting the image input interface and theprocessing unit;

[0069] wherein the processing unit controls the monitoring apparatussuch that:

[0070] each n-th one of the frame images output from the imaging deviceimaging the predetermined monitor area having no object to be detectedis recorded in the memory beforehand as reference background sequentialimages, where n is an integer not less than unity;

[0071] an image from the imaging device imaging the predeterminedmonitor area is sequentially input to the processing unit;

[0072] a reference background image corresponding to the input image isread from the memory; and

[0073] a difference in a pixel value between the input image and thereference background image corresponding to the input image read iscalculated for each pixel, and an area associated with a largedifference value is detected as an intruding object and displayed on themonitor.

[0074] According to a yet further aspect of the invention, there isprovided a monitoring apparatus for monitoring an object intruding intoa predetermined monitor area, comprising:

[0075] an imaging device;

[0076] an image input interface connected to the imaging device forconverting a video signal from the imaging device to image data;

[0077] a processing unit including a central processing unit and amemory for processing the image data;

[0078] a monitor; and

[0079] a bus for interconnecting the image input interface, theprocessing unit and the monitor;

[0080] wherein the processing unit controls the monitoring apparatussuch that:

[0081] each n-th frame image output from the imaging device imaging thepredetermined monitor area having no object to be detected is recordedin the image memory beforehand as reference background sequentialimages, where n is an integer not less than unity;

[0082] images from the imaging device imaging the predetermined monitorarea in accordance with a predetermined scanning pattern aresequentially input to the processing unit;

[0083] reference background sequential images are read from the imagememory in synchronism with the imaging of the input images; and

[0084] a difference in a pixel value between an input image and areference background image read is calculated for each pixel, and thearea associated with a large difference value is detected as anintruding object and displayed on the monitor.

BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescription of the embodiments of the invention as illustrated in theaccompanying drawings wherein:

[0086]FIG. 1 is a diagram for explaining an embodiment of the invention;

[0087]FIG. 2 is a block diagram showing a configuration of an intrudingobject monitoring apparatus according to an embodiment of the invention;

[0088]FIG. 3 is a flowchart showing the processing operation of anobject detecting method according to one embodiment of the invention;

[0089]FIG. 4 is a flowchart showing the processing operation of anobject detecting method according to another embodiment of theinvention;

[0090]FIG. 5 is a flowchart showing the processing operation of anobject detecting method according to still another embodiment of theinvention;

[0091]FIG. 6 is a flowchart showing the processing operation of anobject detecting method according to a further embodiment of theinvention;

[0092]FIGS. 7A, 7B, 7C are diagrams for explaining the displacementbetween an input image and a reference background image;

[0093]FIGS. 8A, 8B a re diagrams for explaining the frame displacementbetween an input image and a reference background image;

[0094]FIG. 9 is a diagram for explaining the principle of the processingaccording to the background image subtraction method;

[0095]FIG. 10 is a flowchart showing a further detailed processingoperation of the step of initializing the reference background imagesequence in the object detecting method shown in FIGS. 3 to 6;

[0096]FIGS. 11A, 11B, 11C, 11D are diagrams for explaining thecorrection of the frame displacement by template matching; and

[0097]FIG. 12 is a diagram for explaining an object detecting methodaccording to a still further embodiment of the invention.

DESCRIPTION OF THE EMBODIMENTS

[0098] Embodiments of the invention will be explained below withreference to the drawings. In all the drawings, similar component partsare designated by similar reference numerals, respectively.

[0099] An intruding object monitoring apparatus according to anembodiment of the invention will be described with reference to FIG. 2.FIG. 2 is a block diagram showing a hardware configuration of anintruding object monitoring apparatus.

[0100] Reference numeral 201 designates a television camera (hereinafterreferred to as the TV camera), numeral 202 a pan and tilt head, numeral203 a zoom lens, numeral 204 an image input I/F, numeral 205 an imageoutput I/F, numeral 216 a monitor, numeral 215 a data bus, numeral 206 acommunication I/F, numeral 213 a pan and tilt head control unit, numeral214 a zoom control unit, numeral 207 an output I/F, numeral 217 an alarmlamp, numeral 208 a CPU (central processing unit), numeral 209 an imagememory, numeral 210 a program memory, numeral 211 a work memory, andnumeral 212 an external storage device. The TV camera 201 is mounted onthe pan and tilt head 202 and includes the zoom lens 203. The pan andtilt head 202 is connected to the pan and tilt head control unit 213.The zoom lens 203 is connected to the zoom control unit 214. The pan andtilt head control unit 213 and the zoom control unit 214 are connectedto the communication I/F 206. The TV camera 201 is connected to theimage input I/F 204. The monitor 216 is connected to the image outputI/F 205. The alarm lamp 217 is connected to the output I/F 207. Also,the image input I/F 204, the image output I/F 205, the communication I/F206, the output I/F 207, the CPU 208, the image memory 209, the programmemory 210, the work memory 211 and the external storage device 212 areconnected to the data bus 215.

[0101] In FIG. 2, the TV camera 201 images a portion of the monitor areaentering the monitor visual field thereof. The intruding objectmonitoring apparatus picks up the image of the whole monitor area byscanning with the TV camera 201 by operating the pan/tilt head 202 andthe zoom lens 303.

[0102] The pan and tilt head 202 changes the imaging direction of the TVcamera 201 by a pan and tilt head control signal from the pan and tilthead control unit 213. The zoom lens 203 changes the zoom ratio of thezoom lens by a zoom control signal of the zoom control unit 214.

[0103] The TV camera 201 converts the image picked up into a videosignal, and the video signal thus converted is input to the image inputI/F 204. The image input I/F converts the input video signal into theimage data of a format (320 pixels wide, 240 pixels tall and 8bits/pixel, for example) handled by the intruding object monitoringapparatus, and the resulting image data are sent to the image memory 209through the data bus 215. The image data thus sent in are stored in theimage memory 209. In accordance with the amount of image data thusstored, the image data are transferred from the image memory 209 to theexternal storage device 212.

[0104] The CPU 208 analyzes the image stored in the image memory 209 inthe work memory 211 in accordance with the program held in the programmemory 210.

[0105] As a result of the analysis described above, if information orrelated information is acquired as to whether an object has intrudedinto the imaging visual field of the TV camera 201 or not, then, inaccordance with the processing result, the CPU 208 causes the pan andtilt head control signal to be transmitted from the data bus 215 throughthe communication I/F 206 to the pan and tilt head control unit 213 onthe one hand and causes the zoom control signal to be transmittedthrough the communication I/F 206 to the zoom control unit 214 on theother.

[0106] Further, the CPU 208 causes a processing result image, forexample, to be displayed on the monitor 216 through the image output I/F205 and turns on the alarm lamp 217 through the output I/F 207. Thecommunication I/F 206 converts the signal from the CPU 208 into a format(RS-232C signal, for example) that can be recognized by the pan and tilthead control unit 213 and the zoom control unit 214, and controls thezoom ratio of the zoom lens 203 and the pan-tilt motor of the pan andtilt head 202.

[0107] On the other hand, the image output I/F 205 converts the signalof the CPU 208 into a format (NTSC video signal, for example) that canbe used by the monitor 216, and sends the resulting signal to themonitor 216. The monitor 216 displays the image of the result ofdetecting an intruding object, for example.

[0108]FIG. 3 is an example of a flowchart showing the processingoperation according to an embodiment of the invention. The processingoperation shown in the flowchart of FIG. 3 is executed by use of theintruding object monitoring apparatus shown in FIG. 2. According to thisfirst embodiment, a reference background image corresponding to theinput image picked up within the monitor visual field is selected fromthe reference background sequential images, i.e. a plurality ofreference background images (reference background moving picture or aplurality of frames of reference background images) stored in the imagememory 209 or the external storage device 212, and in this way an objectthat has intruded into the visual field of the TV camera 201 is detectedby the subtraction method explained with reference to FIG. 9.

[0109] The reference background sequential images or a plurality of thereference background images are obtained by picking up images, asdescribed later, in such a manner that the imaging direction is changedby moving the TV camera 201 or changing the imaging direction thereofusing the pan and tilt head thereof in accordance with a predeterminedscanning pattern while scanning the monitor area having no object to bedetected and thus sequentially changing the monitor visual field of themonitoring apparatus. The “predetermined scanning pattern” is defined asa changing pattern of imaging conditions with time including the changein the predetermined imaging direction and the predetermined trajectorywhen moving the TV camera 201. The predetermined pattern may furtherinclude a changing pattern of a predetermined zoom ratio by which thezoom ratio of the zoom lens of the TV camera 201 is changed with time.In the description that follows, the brief wording “the predeterminedchange of the visual field of the monitoring apparatus” will be used tomean that “the TV camera 201 is moved in accordance with a predeterminedscanning pattern or the imaging direction thereof is changed by the panand tilt head thereof, while scanning the monitor area with the TVcamera 201 thereby to successively change the monitor visual field ofthe monitoring apparatus”.

[0110] Referring to FIG. 3, first, in the reference background movingpicture initialization step 300, the reference background sequentialimages (i.e. the reference background moving picture) is initialized.This process will be explained with reference to FIG. 10. FIG. 10 is aflowchart showing the flow of the process for initializing the referencebackground sequential images. In the process for initializing thereference background sequential images, the step 1001 for determiningthe need of additionally registering the reference background sequentialimages determines whether the reference background sequential imagesstored in the work memory 211 or the external storage device 212 holdsall of the reference background images for the predetermined change ofthe monitor visual field of the monitoring apparatus. In the case wherethe reference background sequential images holds all of the referencebackground images (in the case where the reference background sequentialimages is prepared in advance for the predetermined change of themonitor visual field of the monitoring apparatus), it is determined thatthe additional registration of the reference background sequentialimages is not needed and the process for initializing the referencebackground sequential images is terminated (the process is returned tothe image input step 301). In the case where all of the referencebackground images are not held or only partially held, on the otherhand, it is determined that the additional registration of the referencebackground sequential images is required, and the process branches tothe frame number calculation step 1002. The frame number calculationstep 1002 calculates the frame number of the reference backgroundsequential images to be added. The frame number is used forsynchronization between the input image and the reference backgroundimages held in the reference background sequential images, and the framenumber for the monitor start time is expressed, for example, as frame=0.In the case where the reference background images are held in thereference background sequential images at predetermined samplingintervals (30 frames per second, for example), the frame number of 300,for example, indicates that 10 seconds has passed from the monitor starttime. In other words, the frame number of the reference background imagecorresponding to the input image applied at the time point 10 secondsafter the monitor start time is given as frame=300. In the case wherethe reference background sequential images holds no reference backgroundimage at all, for example, the frame number calculation step 1002 setsthe frame number to 0 expressed as frame=0 (indicating the startingpoint of the predetermined change of the monitor visual field of themonitoring apparatus (corresponding to the monitor visual field at themonitor start time point)). In the case where the reference backgroundsequential images holds ten frames of the reference background images(indicating that ten frames have passed from the starting point of thepredetermined change of the monitor visual field of the monitoringapparatus), it is assumed that frame=10. Then, in the referencebackground image acquisition step 1003, an input image of 320×240pixels, for example, is acquired from the imaging device 201. Further,in the reference background sequential images additional registrationstep 1004, the input image acquired in the reference background imageaquisition step 1003 is registered by being added to the referencebackground sequential images stored in the work memory 211 or theexternal storage device 212. In the additional registration completiondetermining step 1005, on the other hand, the initialization processingfor the reference background sequential images is terminated (byreturning to the image input step 301) in the case where theregistration of all the reference background images by addition to thereference background sequential images is completed for thepredetermined change of the monitor visual field of the monitoringapparatus. In the case where such additional registration is not yetcomplete, on the other hand, the process branches to the frame numbercalculation step 1002.

[0111] Returning to FIG. 3, in the image input step 301, the input videosignal picked up by the TV camera 201 is acquired as an input image of320×240 pixels, for example.

[0112] Then, in the reference background image select/sync read step302, the reference background image 902 having an angle of viewidentical or most similar to the input image 901 obtained in the imageinput step 301 is selected from reference background sequential images(a plurality of the reference background sequential images or thereference moving picture).

[0113] According to this embodiment, in order to select a correspondingreference background image from a plurality of reference backgroundimages (i.e. the reference background sequential images), thesynchronism between the input image and the reference background imagesis managed by the frame number (frame). The frame number of the monitorstart time is determined as frame=0, for example, and the frame numberis increased for each sampling interval of the reference backgroundsequential images in accordance with the time elapsed from the monitorstart time. Specifically, in the case where the reference backgroundsequential images holds the reference background images sampled at therate of 30 frames per second, for example, the frame number increases atthe rate of 30 per second. In the reference background moving pictureinitialization step 300, the frame number of the reference backgroundsequential images is calculated based on the start point of thepredetermined change of the monitor visual field (corresponding to themonitor visual field at monitor start time point) of the monitoringapparatus (1002), the reference background sequential images is acquired(1003), and registered by being added to the reference backgroundsequential images (1004). Take FIG. 12 as an example. The predeterminedchange of the monitor visual field is defined as the change of theimaging visual field from 1202 a, 1202 b, 1202 c, 1202 d, 1202 e, 1202e, 1202 d, 1202 c, 1202 b, 1202 a in that order. With this predeterminedchange of the monitor visual field, the reference background images 0 to9 of the visual field of the nine frame numbers of frames=0 to 9 areheld in the reference background sequential images. At the monitor starttime point, the visual field of the imaging device is 1202 a, and withthe lapse of time, the visual field changes from 1202 a, 1202 b, 1202 c,1202 d, 1202 e, 1202 e, 1202 d, 1202 c, 1202 b, 1202 a in that order,while at the same time increasing the frame number from 0 to 9. As aresult, according to the frame number (frame), the reference backgroundimage acquired in the same visual field as the input image can beselected from the reference background sequential images in thereference background image select step 302. As a result, synchronism canbe achieved between the input image and the reference background imageto be selected.

[0114] In other words, the frame number (frame) is used as a counter forselecting the reference background image corresponding to the inputimage from the reference background sequential images, and increases, asdescribed above, at the rate of 30/sec (when pickedup with a TV cameraconforming to television broadcasting in NTSC system) from the monitorstart time (reference time point) of frame=0. In the case whereframe=300, for example, it indicates that ten seconds has passed fromthe reference time point, and therefore the reference background imageassociated with 10 seconds following the reference time point isselected.

[0115] According to this embodiment, the sampling interval of thereference background image is equal to the frame interval of the videosignal of the TV camera. Nevertheless, the sampling interval can bedetermined arbitrarily without regard to the video signal. In otherwords, every n-th frame image from the TV camera 101 can be used as areference background image, where n is not limited to 1 but may be aninteger of 2 or more. Moreover, the sampling interval is not limited tothe frame interval of the video signal and may be arbitrarily set. Thiswill be described in more detail below.

[0116] According to this embodiment, an example is used in which thevideo signal in NTSC system is sampled for full frames (at the rate of30 frames per second). Nevertheless, the rate of 10 frames per second,for example, can also produce the same effect as the invention. Thelonger the sampling interval, however, the larger the visual fielddisplacement between the input image and the selected referencebackground image, resulting in a more frequent detection error in thesubtraction processing. The time t elapsed from the monitor start time(start time point of the predetermined change of the visual field) andthe frame number (frame) hold the following relation:

t=Δt×frame  (1)

[0117] where Δt is the sampling interval in seconds. In the case wherethe sample operation is performed on the video signal in NTSC system forfull frames, for example, Δt=1/30 second. In the case where the rate is10 frames per second, on the other hand, Δt=1/10 second. Thus, the framenumber can be calculated from the elapsed time t as follows (the numbersbelow decimal point is rounded):

frame=t/Δt  (2)

[0118] According to the method of synchronizing the input image and thereference background image to be selected, the frame number iscalculated from equation (2) above using the time t elapsed from themonitor start time, and the reference background image of the calculatedframe number is selected from the reference background sequentialimages.

[0119] A plurality of the reference background images are stored in theimage memory 209 or the external storage unit 212. The recording formatof a plurality of the reference background images (reference backgroundmoving picture) may be either a succession of reference backgroundimages for a predetermined number of frames or a compressed format suchas MPEG (moving picture experts group) or the motion JPEG (motion jointphotographic experts group).

[0120] In the subtraction processing step 303, the difference of thebrightness value between the input image 901 and the referencebackground image 902 is calculated to produce a difference image 903 foreach pixel.

[0121] In the binarization processing step 304, the difference image 903obtained in the subtraction processing step 303 is processed using apredetermined threshold value Th (=20, for example) in such a mannerthat the brightness value of each pixel less than the threshold value This “0” and the brightness value not less than the threshold value Th is“255” (the brightness value calculated for each pixel in 8 bits) therebyto produce a binarized image 904.

[0122] In the intruding object existence determining step 305, it isdetermined that an intruding object exists in the presence of a clusterof pixels having the brightness value “255” in the binarized image 904.In that case, the process branches to the alarm/monitor display step406. In the case where no such cluster is found, on the other hand, itis determined that there exists no intruding object and the processbranches to the pixel input step 301.

[0123] The process of FIG. 3 will be explained with reference to FIG. 1.FIG. 1 is a diagram for explaining that the reference background imagefor the subtraction method shown in FIG. 9 is selected from a pluralityof reference background images (reference background sequential images).Numeral 101 designates an input image, numeral 102 a referencebackground sequential images, numeral 103 a difference image, numeral104 a binarized image, numeral 105 a reference background imageselector, numeral 106 a subtractor, numeral 107 a binarizer, andnumerals 102A, 102B, 102C, 102D, 102E, 102F, 102G reference backgroundimages included in the reference background sequential images 102. Theinput image 101, the difference image 103, the binarized image 104, thesubtractor 106 and the binarizer 107 are substantially the same as theinput image 901, the difference image 903, the binarized image 904, thesubtractor 905 and the binarizer 906, respectively, described withreference to FIG. 9, and therefore will not be described in detail.

[0124] The reference background sequential images 102 contains inchronological order the reference background images 102A, 102B, 102C,102D, 102E, 102F, 102G for a predetermined change of the monitor visualfield, and the reference background image 102D having substantially thesame angle of view corresponding to the input image 101, for example, isselected by the reference background image selector 105.

[0125] This reference background image selector 105 increases the frames(frame number) by the time the monitor operation has been performed.Even with a scene where the monitor visual field changes, therefore, theproper reference background image can be used and accurate detection ofan intruding object is made possible.

[0126]FIG. 1 shows an example in which the imaging device is mounted atthe head of a carrier moving on a predetermined speed profile along apredetermined track. In the absence of an object to be detected, thereference background sequential images is stored in the image memory 209or the external storage unit 212 in the chronological order ofacquisition as reference background images sampled at predeterminedintervals (30 frames, for example) from the frame images picked up bythe imaging device running along the predetermined track at apredetermined speed.

[0127] As an alternative, however, the imaging device may be fixed inposition and panned and/or tilted (imaging direction) or the setting ofthe zoom ratio of the zoom lens of the imaging device may be changedappropriately.

[0128]FIG. 4 is an example of flowchart for explaining the processingoperation according to a second embodiment of the invention. FIG. 4 is adiagram similar to the flowchart of FIG. 3 to which the referencebackground moving picture update step 401 is added. In FIG. 4, theprocessing operation including the image input step 301 to thebinarization processing step 304 and the alarm/monitor display step 406are already explained with reference to FIG. 3 and will not be explainedfurther. In similar fashion, in the flowcharts explained below, thesteps of the same reference numerals have substantially the samefunction, respectively, and therefore will not be explained.

[0129] In the intruding object existence determining step 305, assumethat a cluster of pixels having the brightness value “255” exists in thebinarized image 904 obtained in the binarization processing step 304.The process proceeds to the alarm/monitor display step 306. In the casewhere it is determined in the intruding object existence determiningstep 305 that the binarized image 904 acquired from the binarizationprocessing step 304 has not any cluster of pixels with the brightnessvalue “255”, on the other hand, the process proceeds to the referencebackground moving picture update step 401.

[0130] The reference background moving picture update step 401 isintended to update the reference background sequential images 102 upondetermination of the intruding object existence determining step 305that there exists no intruding object. Specifically, the referencebackground sequential images 102 can be updated by any method in whichthe input image 101 is reflected, in some form or other, in thereference background image 102D having the same angle of viewcorresponding to the input image 101. For example, the referencebackground image 102D may be replaced with the input image. As anotheralternative, the average value for each pixel of the referencebackground image 102D and the input image 101 is determined, and theimage configured of each average value thus obtained can be used as anew reference background image 102D. In this way, even with a scenehaving a changing monitor visual field, the proper reference backgroundimage can be used while sequentially updating the reference backgroundimage, thereby making it possible to detect an intruding objectaccurately.

[0131]FIG. 5 is an example of flowchart according to a third embodimentof the invention. In FIG. 5, a displacement correcting step 501 isinserted between the reference background image select step 302 and thesubtraction processing step 303 in the flowchart of FIG. 4.

[0132] In the displacement correcting step 501, the amount of positionaldisplacement between the input image 101 and the reference backgroundimage 102D caused by the tremble of the TV camera 201 in movement iscalculated, and based on the calculated displacement amount, theposition of the input image 101 on the screen is corrected. An exampleof the processing will be explained with reference to FIGS. 7A, 7B, 7C.

[0133]FIGS. 7A, 7B, 7C are diagrams for explaining a method ofcalculating the amount of positional displacement utilizing the templatematching according to an embodiment. In FIG. 7A, numeral 701 designatesa reference background image, and numeral 701A a section (block)arranged in the reference background image 701. In FIG. 7B, numeral 702an input image, numeral 702A an area detected as the block 701A in theinput image 702, and numeral 702B an area in the input image 702 at aposition corresponding to the block 701A, and numeral 702C arrowsindicating the amount of positional displacement between the area 702Band the area 702A. In FIG. 7C, numeral 703 shows an image indicating thedistribution of the positional displacement amount for all the blocks inthe reference background image 701.

[0134] In the processing method shown in FIGS. 7A, 7B, 7C, the referencebackground image 701 is divided into several sections (blocks) (eight,in the case under consideration), and it is determined by templatematching at which position the image of each block is located on theinput image.

[0135] In FIGS. 7A, 7B, 7C, the block 701A (the fully hatched area inthe reference background image 701) is shown as an example. The image ofthe block 701A (indicated by the area 702B defined by a dotted line as ablock in the input image 702) is detected as existent in the area 702Ain the input image 702, and the positional displacement thereof isindicated by arrow 702C. In this way, FIG. 7C shows the positionaldisplacement between the input image and the reference background image.Thus, the “positional displacement” or spatial deviation means a spatialdisplacement or deviation (up or down direction, right or leftdirection) of an input image relative to the reference background image.

[0136] The template matching is described in “Introduction to ComputerImage Processing” by Hideyuki Tamura, p.118-125, published by SokenShuppan, 1985, and also disclosed in “Digital Picture Processing” byAzriel Rosenfeld et al., ACADEMIC PRESS, pp.296-303, 1996, and U.S. Pat.No. 5,554,983, the disclosure of which are hereby incorporated byreference herein.

[0137] This template matching processing is carried out for all theblocks thereby to obtain the displacement distribution 703. The averageof this distribution is the displacement amount v of the input image. Inother words, the displacement amount vn of each block is expressed inequation (3) below.

vn=(xn, yn)  (3)

(n=1, 2, . . . , N)

[0138] The displacement amount v of the input image is given byequations (4) and (5).

v=(dx, dy)  (4)

[0139] $\begin{matrix}\left. \begin{matrix}{{x} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{xn}}}} \\{{y} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{yn}}}}\end{matrix} \right\} & (5)\end{matrix}$

[0140] where N is the number (8 in the example taken here) of blocks(sections).

[0141] Then, equation (6) is calculated for v=(dx, dy), where f(x, y)designates an input image, and f′(x, y) an input image of which thedisplacement has been corrected.

f′(x, y)=f(x−dx, y−dy)  (6)

[0142] In this way, the displacement which may exist between the inputimage 101 and the reference background image 102D in a given scene canbe corrected and accurate detection of an intruding object is madepossible.

[0143]FIG. 6 is an example of flowchart according to a fourth embodimentof the invention. The flowchart of FIG. 6 is obtained by adding a framedisplacement detection step 601 and a branching step 602 for branchingthe process to execute the reference background image select step 302again in the presence of a frame displacement are added to the flowchartof FIG. 4.

[0144] The frame displacement detection step 601 is for determining thetime lag between the input image 101 and the selected referencebackground image 102D. An embodiment of the processing will be explainedwith reference to FIGS. 8A, 8B and 11. FIGS. 8A, 8B are diagrams showinga method of determining the frame displacement utilizing the templatematching. In FIG. 8A, numeral 801 designates a reference backgroundimage. In FIG. 8B, numeral 802 designates an image indicating thedistribution of the frame displacement amount for all the blocks inwhich the reference background image 801 is divided.

[0145] The reference background image 801 is divided into several blocksin the left and right parts of the screen (two each in the left andright parts in the embodiment shown in FIGS. 8A, 8B), and as in FIGS.7A, 7B, 7C, the position at which the image of each block is located onthe input image is determined by template matching.

[0146] This determining process will be explained with reference toFIGS. 11A, 11B, 11C, 11D. FIGS. 11A to 11 d show an example assuming thesame scene as in FIG. 1. In FIG. 11A, numeral 1101 designates an inputimage. In FIG. 11B, numeral 1111 designates a selected referencebackground image. The areas 1102, 1103, 1104, 1105 designate the blocksdescribed with reference to FIG. 8A and represent a case in which twoblocks are arranged in each of the left and right parts of the screen asin FIG. 8A. In the reference background image 1111, the areas 1112 a,1113 a, 1114 a, 1115 a defined by dotted lines designate the areascorresponding to the areas 1102, 1103, 1104, 1105, respectively, of theblocks in the input image 1101. The areas 1112 b, 1113 b, 1114 b, 1115 bcan be obtained by template matching using the images of the areas 1102,1103, 1104, 1105, respectively. The position change from the center ofthe areas 1112 a, 1113 a, 1114 a, 1115 a to the center of the areas 1112b, 1113 b, 1114 b, 1115 b, respectively, represents the displacement ofeach block (indicated by arrows 1112 c, 1113 c, 1114 c, 1115 c,respectively). The amount of each of these displacements is expressed asV1=(X1, Y1), V2=(X2, Y2), V3=(X3, Y3), V4=(X4, Y4), respectively. Inthis case, V1, V2 indicate the displacement on the left side of thescreen, and V3, V4 the displacement on right side of the screen.

[0147] Further, from the average displacement amount of the left andright blocks, the average displacement amount of the block on the leftside of the screen is obtained as vL=(xL, yL), while the averagedisplacement amount of the block on the right side of the screen isobtained as vR=(xR, yR).

[0148] Specifically, the average displacement on the left and rightsides of the screen are obtained as xL=(X1+X2)/2, yL=(Y1+Y2)/2,xR=(X2+X4)/2, yR=(Y3+Y4)/3. In the case of FIG. 11 involving-the inputimage 1101 and the reference background image 1111, xL is negative (thedirection from the upper left part to the lower right part of the screenis deemed as positive direction), and xR assumes a positive value (i.e.the arrows 1112 c, 1113 c, 1114 c, 115 c are directed from centeroutward of the screen). This indicates that the reference backgroundimage 1111 is temporally ahead of the input image 1101 (frame is largerthan the proper value). Thus, one frame is reduced in the case wherexR−xL is not less than a predetermined size Tf.

[0149] Now, an opposite example will be explained with reference toFIGS. 11C, 11D. In FIG. 11C, numeral 1121 designates an input image. InFIG. 11D, numeral 1131 designates a selected reference background image.As in the case involving the input image 1101 and the referencebackground image 1111 described above, the areas 1122, 1123, 1124, 1125represent the blocks described with reference to FIG. 8. In thereference background image 1131, the areas 1132 a, 1133 a, 1134 a, 1135a defined by dotted lines correspond to the block areas 1122, 1123,1124, 1125, respectively, in the input image 1121. The areas 1132 b,1133 b, 1134 b, 1135 b, on the other hand, are obtained by templatematching using the images of the areas 1122, 1123, 1124, 1125,respectively. In the process, the position change from the center of theareas 1132 a, 1133 a, 1134 a, 1135 a to the center of the areas 1132 b,1133 b, 1134 b, 1135 b, respectively, indicates the displacement of eachblock (expressed by arrows 1132 b, 1133 b, 1134 b, 1135 b,respectively), and each position displacement amount is expressed asV1=(X1, Y1), V2=(X2, Y2), V3=(X3, Y3), V4=(X4, Y4), respectively. Inthis case, V1, V2 designate the displacement on the left side of thescreen, and V3, V4 the displacement on the right side of the screen.Further, the average displacement on the left and right sides of thescreen is obtained as xL=(X1+X2)/2, yL=(Y1+Y2)/2, xR=(X3+X4)/2,yR=(Y3+Y4)/2. In the case FIG. 11 involving the input image 1121 and thereference background image 1131, xL assumes a positive value and xR anegative value (i.e. the arrows 1112 c, 1113 c, 1114 c, 115 c aredirected to the center inward from outside of the screen). Thisindicates that the reference background image 1131 temporally lagsbehind the input image 1121 (the frame is smaller than the propervalue). In the case where xL−xR is not less than a predeterminedmagnitude Tf, therefore, one frame is increased, where Tf designates anallowable amount of displacement of the pixel caused by the framedisplacement, which value is empirically obtained and set to Tf=5, forexample, in this embodiment.

[0150] According to this embodiment, a total of four blocks, two each onleft and right sides of the screen as in FIG. 8A, are shown as anexample. Nevertheless, any other number of blocks or different numbersof blocks in left and right sides may be arranged with equal effect.

[0151] Then, in the branching step 602, the process branches in such amanner as to execute the reference background image select step 302again in the case where the frame is corrected. By doing so, the framedisplacement can be corrected and accurate detection of an intrudingobject is made possible even in the case where the imaging device ismoving along the optical axis thereof and a time lag exists between theinput image 101 and the reference background image 102D. Thus, the“frame displacement” or temporal deviation means a displacement ordeviation between the proper reference background image which correctlycorresponds to the input image and the reference background imageactually selected by the reference background image selector 105.

[0152] Now, a fifth embodiment of the invention will be explained. Thefifth embodiment of the invention is intended to detect an objectintruding into the visual field of the imaging device while periodicallychanging the imaging direction and the zoom ratio of the TV camera 201by the pan and tilt head 202 and the zoom lens 203.

[0153] Specifically, in the absence of an intruding object to bedetected, the pan and tilt head 202 and the zoom lens 203 are controlledby a control signal so that the imaging direction and the zoom ratio ofthe zoom lens 203 of the TV camera 201 are changed by an amountequivalent to one period, and the resulting input image is contained inthe reference background image and the reference background sequentialimages 102.

[0154] The reference background image selector 105 resets the frame to 0upon completion of a period of monitor operation.

[0155] This process will be explained with reference to FIG. 12. FIG. 12shows a case in which the viewing direction of the TV camera 1201 ischanged from frame number 0 to 9. In FIG. 12, for facilitating theunderstanding, the change only in the viewing direction is displayed,and an example is shown in which the imaging visual field (input image)is periodically changed in the order of 1202 a, 1202 b, 1202 c, 1202 d,1202 e. At the monitor start time point (frame=0), the imaging visualfield is 1202 a, which changes to 1202 b, 1202 c, 1202 d, 1202 e, 1202e, 1202 d, 1202 c, 1202 b, 1202 a with the progress of the monitorprocessing. At the same time, the frame also changes from 0 to 9.According to a fifth embodiment of the invention, the frame number isset to 0 when the imaging visual field returns to 1202 a. By doing so,even in the case where the frame displacement develops when changing theviewing direction (in the case where the input image and the referencebackground image go out of phase), the input image and the referencebackground image can be synchronized with each other at the time pointwhen the imaging visual field returns to 1202 a. Also, a specificpattern like a mark such as a cross 1203 is preset in the imaging visualfield 1202 c, for example, and the very instant the particular patternof the mark is displayed in the input image, the frame number iscorrected to a predetermined value (in the case of FIG. 12, frame is setto 2 when the imaging visual field is directed from 1202 a to 1202 e,and frame is set to 7 when the imaging visual field is directed from1202 e to 1202 a). By doing so, even in the case where the framedisplacement occurs, the frame number can be corrected to a propervalue, the instant the pattern providing the mark is displayed. Thiscorrection can be realized by at least one of the indexes including theposition information of the imaging device (the information such as apredetermined reference position in the aforementioned case) and theimaging visual field information (the information such as a patternproviding a specific mark in the aforementioned case).

[0156] Thus, according to this invention, even in the case where thesetting of the pan and tilt head 202 and the zoom lens 203 areperiodically changed, the proper reference background image can beobtained and an intruding object can be accurately detected.

[0157] According to a sixth embodiment of the invention, the imagingdevice is mounted on a vehicle such as a train running on apredetermined track, and the input image obtained in the absence of anintruding object is contained as a reference background image in areference background sequential images 102. According to this invention,therefore, even in the case where the imaging device is mounted on avehicle running on a predetermined track, the proper referencebackground image can be obtained and an intruding object can beaccurately detected.

[0158] The flowcharts of FIGS. 3 to 6 showing the embodiments refer tothe case in which the processing operation for detecting an object or anintruding object is not terminated midway. In the hardware configurationof the intruding object monitoring apparatus described with reference toFIG. 2, however, it is apparent that the processing operation isterminated midway upon termination of the operation of the monitoringapparatus due to an incident such as power failure or at the will of theuser.

[0159] Upon termination of the processing operation, the detectionresult so far obtained and the reference background image may be held inan arbitrary storage unit such as a (nonvolatile) work memory or anexternal memory to enable them to be effectively reused at thesubsequent time of the operation restart.

[0160] As described above, according to this embodiment, a referencebackground sequential images containing the reference background imagesof two or more frames is stored against such a predetermined change ofthe imaging position as a change of the imaging direction or the zoomratio of the imaging device, and when the difference of brightness iscalculated, the proper reference background image is used from thereference background sequential images. In this way, even in the casewhere the imaging position changes accompanied by a change in thesetting of the zoom ratio of the zoom lens or the imaging direction ofthe imaging device, an intruding object located in the imaging visualfield can be detected, thereby greatly widening the range of applicationof the intruding object detecting apparatus.

[0161] For example, the intruding object detecting apparatus accordingto the invention can be mounted on a mobile unit (such as a train or apan-tilt camera whose moving route is determined) to detect an objectusing the image taken by the camera which undergoes a constant change ofthe visual field.

1. An object detecting method for detecting an object in a predeterminedmonitor area, comprising the steps of: imaging a plurality of differentareas in said predetermined monitor area having no object at apredetermined time by an imaging device and registering respectiveimages corresponding to said different areas having no image of saidobject to be detected; at a different time from said predetermined time,imaging said predetermined monitor area by said imaging device;comparing an image from said imaging device with a corresponding one ofsaid plurality of registered images; and detecting said object to bedetected based on a result of the comparison.
 2. A method according toclaim 1, wherein said plurality of registered images are referencebackground sequential images and said step of comparing is subtractionprocessing between said image from said imaging device and saidcorresponding one of said reference background sequential images.
 3. Amethod according to claim 2, further comprising the step of detecting adisplacement between the image from the imaging device and acorresponding reference background image, then correcting the image fromsaid imaging device in accordance with the detected displacement,wherein said subtraction processing is carried out between the correctedimage and the corresponding reference background image.
 4. A methodaccording to claim 3, wherein said step of detecting the displacementincludes the step of applying a template matching between said imagefrom said imaging device and the corresponding reference backgroundimage to detect said displacement.
 5. A method according to claim 4,wherein said template matching step is such that the referencebackground image is divided into a plurality of sections, wherein saidimage from said imaging device is subjected to said template matchingusing the image of each section as a template, and wherein an average ofdisplacements detected is used as said displacement.
 6. A methodaccording to claim 2, further comprising the step of detecting a framedisplacement between the frame of the image from the imaging device andthe frame of said corresponding reference background image, wherein whenthere is any frame displacement, a different reference background imageis selected.
 7. A method according to claim 6, wherein said framedisplacement detecting step includes the step of applying a templatematching between said image from said imaging device and saidcorresponding reference background image and detecting a framedisplacement.
 8. A method according to claim 7, wherein said templatematching step is such that said corresponding reference background imageis divided into a plurality of sections, and using the image of eachsection as a template, the template matching is carried out with saidimage from said imaging device, and wherein in accordance with framedisplacement information detected, a reference background image of theframe temporally ahead of or lagging behind the frame of said referencebackground image is selected.
 9. A method according to claim 6, whereinsaid frame displacement detecting step corrects the frame displacementbased on at least one of a position and imaging visual field informationof said imaging device.
 10. A method according to claim 9, wherein aspecific position of the imaging device and a specific frame of thereference background image corresponding to said specific position areset in association with each other in advance, and when the imagingdevice arrives at said specific position, the frame displacement iscorrected using the reference background image of said specific frame.11. A method according to claim 9, wherein said imaging visual fieldinformation contains a specific object within the predetermined monitorvisual field as a mark, and said mark and the reference background imageof a specific frame corresponding to said mark are set in associationwith each other in advance, and wherein when the imaging device picks upthe image of said mark, the frame displacement is corrected using thereference background image of the specific frame.
 12. A method accordingto claim 2, further comprising the step of updating the referencebackground sequential images to update at least one of the referencebackground sequential images.
 13. A method according to claim 12,wherein said update step functions in such a manner that when an objectis not detected in said image from said imaging device in said objectdetection processing step, said corresponding reference background imageis updated with said image.
 14. An object detecting method for detectingan object in a predetermined monitor area, comprising the steps of:imaging a plurality of different areas in the predetermined monitor areaat a predetermined time by the imaging device in accordance with apredetermined scanning pattern, and registering respective referencebackground sequential images corresponding to said different areas in astorage device; at a different time from said predetermined time,imaging said predetermined monitor area by said imaging devicesubstantially in accordance with said predetermined scanning pattern insynchronism with the operation of reading said registered referencebackground sequential images from the storage device; carrying out thesubtraction processing between the images from the imaging device andthe reference background sequential images read out; and carrying outobject detection processing based on a result of the subtractionprocessing.
 15. A method according to claim 14, wherein saidpredetermined scanning pattern is a predetermined chronological changeof at least one or a combination of two or more of the zoom ratio of azoom lens of said imaging device, an imaging direction and a track alongwhich said imaging device moves.
 16. A method according to claim 14,wherein said predetermined scanning pattern is such that said imagingdevice moves on a predetermined speed profile along a predeterminedmoving track.
 17. A method according to claim 14, wherein saidpredetermined scanning pattern is such that the position of the imagingdevice is fixed while the zoom ratio and the imaging direction of theimaging device periodically change.
 18. A method according to claim 14,wherein said reference background sequential images are a set of imagesobtained by scanning said predetermined monitor area having no objectwith the imaging device according to the predetermined scanning patternand sampling the images from said imaging device at predeterminedsampling intervals, wherein said registration step is such that eachimage of said reference background sequential images is registered bybeing assigned a frame number in order of sampling, and wherein saidimaging step is such that the imaging operation of the imaging deviceand the operation of reading the reference background image from thestorage device are carried out in synchronism with each other using theframe number, so that a reference background image corresponding to saidimage from the imaging device is selected from the reference backgroundsequential images.
 19. A method according to claim 18, wherein the framenumber of the corresponding reference background image is calculatedfrom a relation between the time elapsed from start of monitoring to thepresent point in time and the predetermined sampling interval, andwherein the imaging operation of the imaging device and the operation ofreading the reference background image from the storage device aresynchronized with each other using said calculated frame number.
 20. Amethod according to claim 19, further comprising the step of detecting aframe displacement between the image from the imaging device and saidselected reference background image, wherein in the presence of a framedisplacement, another reference background image is selected.
 21. Amethod according to claim 20, wherein said frame displacement detectionstep includes the template matching step for detecting the framedisplacement by template matching between said image from said imagingdevice and said selected reference background image.
 22. A methodaccording to claim 21, wherein said template matching step includes thesubsteps of: dividing said selected reference background image into aplurality of sections; carrying out the template matching on said imagefrom said imaging device using the image of each of said section as atemplate; and selecting a reference background image temporally ahead ofor lagging behind said selected reference background image in accordancewith the detected displacement information.
 23. A method according toclaim 20, wherein said frame displacement detection step is such thatsaid frame displacement is corrected based on at least one of a positionof said imaging device and imaging visual field information.
 24. Amethod according to claim 23, wherein a specific position of saidimaging device and a specific frame number of the reference backgroundimage corresponding to said specific position are set in associationwith each other in advance, and when said imaging device reaches saidspecific position, said frame displacement is corrected using saidspecific frame number.
 25. A method according to claim 23, wherein saidimaging visual field information contains a specific object locatedwithin said predetermined monitor area as a mark, wherein said mark anda specific frame number of the reference background image correspondingto said mark are set in association with each other in advance, andwherein when said imaging device picks up the image of said mark, saidframe displacement is corrected using said specific frame number.
 26. Amethod according to claim 14, further comprising the step of detecting aspatial displacement between said image from said imaging device andsaid reference background image that has been read, and correcting saidread image in accordance with said detected spatial displacement, saidsubtraction processing being carried out using said corrected image. 27.An object detecting apparatus for detecting an object located within apredetermined monitor area, comprising: an imaging device; an imageinput interface connected to said imaging device for converting a videosignal of the imaging device to image data; a processing unit includinga central processing unit and a memory for processing said image data;and a bus for interconnecting said image input interface and saidprocessing unit; wherein in order to detect the object, said processingunit controls said object detecting apparatus such that: images fromsaid imaging device imaging said predetermined monitor area having noobject to be detected are stored sequentially in said memory; an imagefrom said imaging device imaging said predetermined monitor area inaccordance with a predetermined scanning pattern is sequentiallyinputted to said processing unit; an image having no object to bedetected corresponding to said inputted image is read out of saidmemory; and said input image and said read image having no object to bedetected are compared and based on a result of comparison, objectdetection processing is carried out.
 28. An apparatus according to claim27, wherein said frame images having no image of said object to bedetected are reference background sequential images and said comparingincluding subjecting said image from said imaging device and saidcorresponding image having no image of said object to be detected tosubtraction processing.
 29. An apparatus according to claim 27, furthercomprising a zoom lens control unit connected to said bus for changing azoom ratio of a zoom lens of said imaging device, and a pan and tilthead control unit for changing an imaging direction of said imagingdevice.
 30. An apparatus according to claim 27, wherein said imagingdevice is mounted on a moving unit.
 31. An apparatus according to claim30, wherein said moving unit includes a mobile unit.
 32. An apparatusaccording to claim 30, wherein said moving device includes a pan andtilt head.
 33. An object detecting apparatus for detecting an objectwithin a predetermined monitor area, comprising: an imaging device; animage input interface connected to the imaging device for converting avideo signal from the imaging device to image data; a processing unitincluding a central processing unit and a memory for processing theimage data; and a bus for interconnecting the image input interface andthe processing unit; wherein in order to detect the object, saidprocessing unit controls said detecting apparatus such that: an imagefrom said imaging device imaging said predetermined monitor area havingno object to be detected is recorded in said memory beforehand as areference background image of reference background sequential images; animage from the imaging device imaging the predetermined monitor area inaccordance with a predetermined scanning pattern is sequentiallyinputted to the processing unit; a reference background image is readout of said reference background sequential images from the image memoryin synchronism with picking up the input frame images; and a differencein a pixel value between the input image and the read referencebackground image is calculated for each pixel, and an area associatedwith a large difference value is detected as an object.
 34. A monitoringapparatus for monitoring an object intruding into a predeterminedmonitor area, comprising: an imaging device; an image input interfaceconnected to the imaging device for converting a video signal from theimaging device to image data; a processing unit including a centralprocessing unit and a memory for processing said image data; a monitor;and a bus for interconnecting said image input interface and saidprocessing unit; wherein said processing unit controls said monitoringapparatus such that: each n-th image from the imaging device imaging thepredetermined monitor area having no object to be detected is recordedin the memory beforehand as reference background sequential images,where n is an integer not less than unity; an image from the imagingdevice imaging the predetermined monitor area is sequentially input tothe processing unit; a reference background image corresponding to theinput image is read from the memory; and a difference in a pixel valuebetween the input image and the reference background image correspondingto the input image read is calculated for each pixel, and an areaassociated with a large difference value is detected as an object anddisplayed on said monitor.
 35. A monitoring apparatus for monitoring anobject intruding into a predetermined monitor area, comprising: animaging device; an image input interface connected to said imagingdevice for converting a video signal from the imaging device to imagedata; a processing unit including a central processing unit and a memoryfor processing said image data; a monitor; and a bus for interconnectingthe image input interface, the processing unit and the monitor; whereinsaid processing unit controls said monitoring apparatus such that: eachn-th image from the imaging device imaging the predetermined monitorarea having no object to be detected is recorded in the memorybeforehand as reference background sequential images, where n is aninteger not less than unity; images from the imaging device imaging thepredetermined monitor area in accordance with a predetermined scanningpattern are sequentially input to the processing unit; referencebackground sequential images are read from the memory in synchronismwith the operation of picking up the input images; and a difference in apixel value between an input image and a reference background image readis calculated for each pixel, and an area associated with a largedifference value is detected as an object and displayed on the monitor.